Ensemble Nyström

A common problem in many areas of large-scale machine learning involves manipulation of a large matrix. This matrix may be a kernel matrix arising in Support Vector Machines [9, 15], Kernel Principal Component Analysis [47] or manifold learning [43,51]. Large matrices also naturally arise in other applications, e.g., clustering, collaborative filtering, matrix completion, and robust PCA. For these largescale problems, the number of matrix entries can easily be in the order of billions or more, making them hard to process or even store. An attractive solution to this problem involves the Nyström method, in which one samples a small number of columns from the original matrix and generates its low-rank approximation using the sampled columns [53]. The accuracy of the Nyström method depends on the number columns sampled from the original matrix. Larger the number of samples, higher the accuracy but slower the method. In the Nyström method, one needs to perform SVD on a l × l matrix where l is the number of columns sampled from the original matrix. This SVD operation is typically carried out on a single machine. Thus, the maximum value of l used for an application is limited by the capacity of the machine. That is why in practice, one restricts l to be less than 20K or 30K, even when the size of matrix is in millions. This restricts the accuracy of the Nyström method in very large-scale settings. This chapter describes a family of algorithms based on mixtures of Nyström approximations called, Ensemble Nyström algorithms, which yields more accurate low-rank approximations than the standard Nyström method. The core idea of Ensemble Nyström is to sample many subsets of columns from the original matrix, each containing a relatively small number of columns. Then, Nyström method is